Performance Characteristics of HYDRA - A Multi-physics Simulation Code from LLNL

HYDRA simulates a variety of experiments carried out at the National Ignition Facility and other high energy density physics facilities. It has packages to simulate radiation transfer, atomic physics, hydrodynamics, laser propagation, and a number of other physics effects. HYDRA has over one million lines of code, includes MPI and thread-level (OpenMP and pthreads) parallelism, has run on a variety of platforms for two decades, and is undergoing active development. In this paper, we demonstrate that HYDRA’s thread-based load balancing approach is very effective. Hardware counters from IBM Blue Gene/Q runs show that none of HYDRA’s packages are memory bandwidth limited, a few come close to the maximum integer instruction issue rate, and all are well below the maximum floating point issue rate. Topics: Large-scale Simulations in CS&E, Multiscale and Multiphysics Problems, Performance Analysis. 1 HYDRA A Multi-Physics Simulation Code The goal of this paper is to introduce readers to a complex “multi-physics” code, discuss some of the techniques used to improve performance, and use data from hardware counters to provide insight into the bottlenecks controlling the performance. We chose HYDRA [1] [2], which is used to simulate experiments conducted at the National Ignition Facility (NIF) [3] and other pulsed laser facilities, as our test code. The laser deposits a large amount of energy in a small volume, so HYDRA is focused on simulating the processes of high energy density physics. HYDRA is a ”multi-physics” simulation code. Figure 1 shows the many physics packages in HYDRA and their interconnections. HYDRA has characteristics similar to other multi-physics codes at LLNL. It consists of over a million lines of code, has run on a variety of platforms for two decades, and is still undergoing active development. HYDRA runs a wide range of simulations and only a subset of the physics packages are used in any given run. This is the first paper to present a performance analysis of HYDRA. Singlephysics codes may have a single loop which consumes over 90% of the run time. Physical processes modeled by the HYDRA code for ICF simulations Author—NIC Review, December 2011 1 NIF-0000-00000s2.ppt Laser light Magnetic fields Burn products 3D ray tracing Spherical DD raytrace 3D MHD Resistive General circuit model TN reactions Multi-group diffusion CP Free streaming neutron transport Monte Carlo transport of neutrons, gammas, charged particles Ion beams 3D ray tracing Monte Carlo